Mobile terminal and method of operating thereof

ABSTRACT

A mobile terminal includes a camera, a display configured to display a real image projected by the camera, and a controller configured to recognize the real image to request a traveling path corresponding to the real image from a robot cleaner, to generate an augmented reality (AR) image of the traveling path based on the traveling path received from the robot cleaner, and to display the AR image to be superimposed on the real image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. 119 and 365 to KoreanPatent Application No. 10-2016-0110737, filed on Aug. 30, 2016 in theKorean Intellectual Property Office, the disclosure of which isincorporated herein by reference.

FIELD

The present invention relates to a mobile terminal and a method ofoperating thereof, and more particularly, to a mobile terminal fordisplaying a traveling path of a robot cleaner to a user through anaugmented reality (AR) image, and a method of operating the same.

BACKGROUND

The functions of a mobile terminal have gradually been diversified.Recently, the mobile terminal may apply augmented reality to display animage. Augmented reality (AR) is technology of superimposing a virtualimage on a real image or background to display one image.

The mobile terminal may display a virtual image related to a real imageto be superimposed on the real image projected by a camera mountedtherein. The mobile terminal may provide a new interesting userexperience (UX) to a user such that the user may easily acquireinformation related to the real image.

Meanwhile, various types of electronic devices for increasing userconvenience are located in a living space of a user. A robot cleaner asan electronic device performs cleaning while autonomously traveling inan indoor space.

In general, the robot cleaner may display a traveling path as a cleaningmap. In this case, an image of a real space and an image of a cleaningmap are simultaneously displayed such that a user checks a region inwhich cleaning is performed. In this case, since the two images aredisplayed with no processing applied, the two images are mixed, therebydecreasing readability. Therefore, the user cannot intuitively andclearly recognize the real space corresponding to the traveling path ofthe robot cleaner.

SUMMARY

An object of the present invention is to display a traveling path usingan augmented reality (AR) image such that a user intuitively and clearlyrecognizes a real space corresponding to the traveling path of a robotcleaner.

Another object of the present invention is to display an uncleanedregion using an AR image such that a user intuitively and clearlyrecognizes a real space corresponding to the uncleaned region.

According to an aspect of the present invention, a mobile terminalincludes a display configured to display a real image projected by abuilt-in camera, and a controller configured to recognize the real imageto request a traveling path corresponding to the real image from a robotcleaner, to generate an augmented reality (AR) image of the travelingpath based on the traveling path received from the robot cleaner, and todisplay the AR image to be superimposed on the real image. Therefore, auser can intuitively and clearly recognize a real space corresponding tothe traveling path of the robot cleaner.

The AR image of the traveling path may include at least one of a cleanedregion and an uncleaned region. Therefore, a user can intuitively andclearly recognize a real space corresponding to the uncleaned region.

According to another aspect of the present invention, an augmentedreality system includes a robot cleaner configured to perform cleaningwhile automatically traveling and a mobile terminal including acontroller configured to recognize a real image projected by a camera torequest a traveling path corresponding to the real image from a robotcleaner, to generate an augmented reality (AR) image of the travelingpath based on the traveling path received from the robot cleaner, and todisplay the AR image to be superimposed on the real image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a mobile terminal in accordance with thepresent disclosure.

FIG. 2 is a perspective view illustrating one example of a glass-typemobile terminal according to another exemplary embodiment.

FIG. 3 is a diagram illustrating the configuration of an augmentedreality system according to an embodiment of the present invention.

FIGS. 4A to 4C are diagrams illustrating the configuration of a robotcleaner according to an embodiment of the present invention.

FIG. 5 is a block diagram showing the configuration of a robot cleaneraccording to an embodiment of the present invention.

FIGS. 6A and 6B are diagrams illustrating the case where a mobileterminal according to an embodiment of the present invention operates inan AR mode.

FIGS. 7A and 7B are diagrams showing a cleaning map displayed by ageneral robot cleaner.

FIG. 8 is a diagram illustrating an example in which a mobile terminalaccording to an embodiment of the present invention maps a real space toa virtual space.

FIGS. 9A to 9D are diagrams illustrating an example in which a mobileterminal according to an embodiment of the present invention displays atraveling path of a robot cleaner using an AR image.

FIGS. 10A to 10D are diagram illustrating an example in which a mobileterminal according to an embodiment of the present invention displays anuncleaned region of a robot cleaner using an AR image.

FIG. 11 is a diagram showing a process of operating a mobile terminalaccording to an embodiment of the present invention.

FIG. 12 is a diagram showing a process of operating a mobile terminalaccording to another embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Description will now be given in detail according to exemplaryembodiments disclosed herein, with reference to the accompanyingdrawings. In general, a suffix such as “module” and “unit” may be usedto refer to elements or components. Use of such a suffix herein ismerely intended to facilitate description of the specification, and thesuffix itself is not intended to give any special meaning or function.

Terminals presented herein may be implemented using a variety ofdifferent types of terminals. Examples of such terminals includecellular phones, smart phones, laptop computers, digital broadcastterminals, personal digital assistants (PDAs), portable multimediaplayers (PMPs), navigators, slate PCs, tablet PCs, ultra books, wearabledevices (for example, smart watches, smart glasses, head mounteddisplays (HMDs)), internet of things (IoT) terminal, and the like.

By way of non-limiting example only, further description will be madewith reference to particular types of mobile terminals. However, suchteachings apply equally to other types of terminals, such as those typesnoted above. In addition, these teachings may also be applied tostationary terminals such as desktop computers, and the like.

FIG. 1 is a block diagram of a mobile terminal in accordance with thepresent disclosure.

Referring to FIG. 1, the mobile terminal 100 is shown having componentssuch as a wireless communication unit 110, an input unit 120, a sensingunit 140, an output unit 150, an interface unit 160, a memory 170, acontroller 180, and a power supply unit 190. It is understood thatimplementing all of the illustrated components is not a requirement, andthat greater or fewer components may alternatively be implemented.

The wireless communication unit 110 typically includes one or moremodules which permit communications such as wireless communicationsbetween the mobile terminal 100 and a wireless communication system,communications between the mobile terminal 100 and another terminal,communications between the mobile terminal 100 and an external server.Further, the wireless communication unit 110 typically includes one ormore modules which connect the mobile terminal 100 to one or morenetworks. To facilitate such communications, the wireless communicationunit 110 includes one or more of a broadcast receiving module 111, amobile communication module 112, a wireless Internet module 113, ashort-range communication module 114, and a location information module115.

The input unit 120 includes a camera 121 for obtaining images or video,a microphone 122, which is one type of audio input device for inputtingan audio signal, and a user input unit 123 (for example, a touch key, apush key, a mechanical key, a soft key, and the like) for allowing auser to input information. Data (for example, audio, video, image, andthe like) is obtained by the input unit 120 and may be analyzed andprocessed by controller 180 according to device parameters, usercommands, and combinations thereof.

The sensing unit 140 is typically implemented using one or more sensorsconfigured to sense internal information of the mobile terminal, thesurrounding environment of the mobile terminal, user information, andthe like. For example, the sensing unit 140 may include at least one ofa proximity sensor 141, illumination sensor 142, a touch sensor, anacceleration sensor, a magnetic sensor, a G-sensor, a gyroscope sensor,a motion sensor, an RGB sensor, an infrared (IR) sensor, a finger scansensor, a ultrasonic sensor, an optical sensor (for example, camera121), a microphone 122, a battery gauge, an environment sensor (forexample, a barometer, a hygrometer, a thermometer, a radiation detectionsensor, a thermal sensor, and a gas sensor, among others), and achemical sensor (for example, an electronic nose, a health care sensor,a biometric sensor, and the like). The mobile terminal 100 may beconfigured to utilize information obtained from sensing unit 140, and inparticular, information obtained from one or more sensors of the sensingunit 140, and combinations thereof.

The output unit 150 is typically configured to output various types ofinformation, such as audio, video, tactile output, and the like. Theoutput unit 150 is shown having a display unit 151, an audio outputmodule 152, a haptic module 153, and an optical output module 154.

The display unit 151 may have an inter-layered structure or anintegrated structure with a touch sensor in order to facilitate a touchscreen. The touch screen may provide an output interface between themobile terminal 100 and a user, as well as function as the user inputunit 123 which provides an input interface between the mobile terminal100 and the user.

The interface unit 160 serves as an interface with various types ofexternal devices that can be coupled to the mobile terminal 100. Theinterface unit 160, for example, may include any of wired or wirelessports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,and the like. In some cases, the mobile terminal 100 may performassorted control functions associated with a connected external device,in response to the external device being connected to the interface unit160.

The memory 170 is typically implemented to store data to support variousfunctions or features of the mobile terminal 100. For instance, thememory 170 may be configured to store application programs executed inthe mobile terminal 100, data or instructions for operations of themobile terminal 100, and the like. Some of these application programsmay be downloaded from an external server via wireless communication.Other application programs may be installed within the mobile terminal100 at time of manufacturing or shipping, which is typically the casefor basic functions of the mobile terminal 100 (for example, receiving acall, placing a call, receiving a message, sending a message, and thelike). It is common for application programs to be stored in the memory170, installed in the mobile terminal 100, and executed by thecontroller 180 to perform an operation (or function) for the mobileterminal 100.

The controller 180 typically functions to control overall operation ofthe mobile terminal 100, in addition to the operations associated withthe application programs. The controller 180 may provide or processinformation or functions appropriate for a user by processing signals,data, information and the like, which are input or output by the variouscomponents depicted in FIG. 1, or activating application programs storedin the memory 170. As one example, the controller 180 controls some orall of the components illustrated in FIG. 1 according to the executionof an application program that have been stored in the memory 170.

The power supply unit 190 can be configured to receive external power orprovide internal power in order to supply appropriate power required foroperating elements and components included in the mobile terminal 100.The power supply unit 190 may include a battery, and the battery may beconfigured to be embedded in the terminal body, or configured to bedetachable from the terminal body.

Referring still to FIG. 1, various components depicted in this figurewill now be described in more detail. Regarding the wirelesscommunication unit 110, the broadcast receiving module 111 is typicallyconfigured to receive a broadcast signal and/or broadcast associatedinformation from an external broadcast managing entity via a broadcastchannel. The broadcast channel may include a satellite channel, aterrestrial channel, or both. In some embodiments, two or more broadcastreceiving modules 111 may be utilized to facilitate simultaneouslyreceiving of two or more broadcast channels, or to support switchingamong broadcast channels.

The broadcast managing entity may be a server which generates andtransmits a broadcast signal and/or broadcast associated information, ora server which receives a pre-generated broadcast signal and/orbroadcast associated information, and sends such items to the mobileterminal.

The broadcast signal may be implemented using any of a TV broadcastsignal, a radio broadcast signal, a data broadcast signal, andcombinations thereof, among others. The broadcast signal in some casesmay further include a data broadcast signal combined with a TV or radiobroadcast signal.

The broadcast signal may be encoded according to any of a variety oftechnical standards or broadcasting methods (for example, InternationalOrganization for Standardization (ISO), International ElectrotechnicalCommission (IEC), Digital Video Broadcast (DVB), Advanced TelevisionSystems Committee (ATSC), and the like) for transmission and receptionof digital broadcast signals. The broadcast receiving module 111 canreceive the digital broadcast signals using a method appropriate for thetransmission method utilized.

Examples of broadcast associated information may include informationassociated with a broadcast channel, a broadcast program, a broadcastevent, a broadcast service provider, or the like. The broadcastassociated information may also be provided via a mobile communicationnetwork, and in this case, received by the mobile communication module112.

The broadcast associated information may be implemented in variousformats. For instance, broadcast associated information may include anElectronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB),an Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld(DVB-H), and the like. Broadcast signals and/or broadcast associatedinformation received via the broadcast receiving module 111 may bestored in a suitable device, such as a memory 170.

The mobile communication module 112 can transmit and/or receive wirelesssignals to and from one or more network entities. Typical examples of anetwork entity include a base station, an external mobile terminal, aserver, and the like. Such network entities form part of a mobilecommunication network, which is constructed according to technicalstandards or communication methods for mobile communications (forexample, Global System for Mobile Communication (GSM), Code DivisionMulti Access (CDMA), CDMA2000 (Code Division Multi Access 2000), EV-DO(Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WidebandCDMA (WCDMA), High Speed Downlink Packet access (HSDPA), HSUPA (HighSpeed Uplink Packet Access), Long Term Evolution (LTE), LTE-A (Long TermEvolution-Advanced), and the like). Examples of wireless signalstransmitted and/or received via the mobile communication module 112include audio call signals, video (telephony) call signals, or variousformats of data to support communication of text and multimediamessages.

The wireless Internet module 113 is configured to facilitate wirelessInternet access. This module may be internally or externally coupled tothe mobile terminal 100. The wireless Internet module 113 may transmitand/or receive wireless signals via communication networks according towireless Internet technologies.

Examples of such wireless Internet access include Wireless LAN (WLAN),Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Digital Living Network Alliance(DLNA), Wireless Broadband (WiBro), Worldwide Interoperability forMicrowave Access (WiMAX), High Speed Downlink Packet Access (HSDPA),HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE),LTE-A (Long Term Evolution-Advanced), and the like. The wirelessInternet module 113 may transmit/receive data according to one or moreof such wireless Internet technologies, and other Internet technologiesas well.

In some embodiments, when the wireless Internet access is implementedaccording to, for example, WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE,LTE-A and the like, as part of a mobile communication network, thewireless Internet module 113 performs such wireless Internet access. Assuch, the Internet module 113 may cooperate with, or function as, themobile communication module 112.

The short-range communication module 114 is configured to facilitateshort-range communications. Suitable technologies for implementing suchshort-range communications include BLUETOOTH™, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), andthe like. The short-range communication module 114 in general supportswireless communications between the mobile terminal 100 and a wirelesscommunication system, communications between the mobile terminal 100 andanother mobile terminal 100, or communications between the mobileterminal and a network where another mobile terminal 100 (or an externalserver) is located, via wireless area networks. One example of thewireless area networks is a wireless personal area networks.

In some embodiments, another mobile terminal (which may be configuredsimilarly to mobile terminal 100) may be a wearable device, for example,a smart watch, a smart glass or a head mounted display (HMD), which isable to exchange data with the mobile terminal 100 (or otherwisecooperate with the mobile terminal 100). The short-range communicationmodule 114 may sense or recognize the wearable device, and permitcommunication between the wearable device and the mobile terminal 100.In addition, when the sensed wearable device is a device which isauthenticated to communicate with the mobile terminal 100, thecontroller 180, for example, may cause transmission of data processed inthe mobile terminal 100 to the wearable device via the short-rangecommunication module 114. Hence, a user of the wearable device may usethe data processed in the mobile terminal 100 on the wearable device.For example, when a call is received in the mobile terminal 100, theuser may answer the call using the wearable device. Also, when a messageis received in the mobile terminal 100, the user can check the receivedmessage using the wearable device.

The location information module 115 is generally configured to detect,calculate, derive or otherwise identify a position of the mobileterminal. As an example, the location information module 115 includes aGlobal Position System (GPS) module, a Wi-Fi module, or both. Ifdesired, the location information module 115 may alternatively oradditionally function with any of the other modules of the wirelesscommunication unit 110 to obtain data related to the position of themobile terminal.

As one example, when the mobile terminal uses a GPS module, a positionof the mobile terminal may be acquired using a signal sent from a GPSsatellite. As another example, when the mobile terminal uses the Wi-Fimodule, a position of the mobile terminal can be acquired based oninformation related to a wireless access point (AP) which transmits orreceives a wireless signal to or from the Wi-Fi module.

The input unit 120 may be configured to permit various types of input tothe mobile terminal 120. Examples of such input include audio, image,video, data, and user input. Image and video input is often obtainedusing one or more cameras 121. Such cameras 121 may process image framesof still pictures or video obtained by image sensors in a video or imagecapture mode. The processed image frames can be displayed on the displayunit 151 or stored in memory 170. In some cases, the cameras 121 may bearranged in a matrix configuration to permit a plurality of imageshaving various angles or focal points to be input to the mobile terminal100. As another example, the cameras 121 may be located in astereoscopic arrangement to acquire left and right images forimplementing a stereoscopic image.

The microphone 122 is generally implemented to permit audio input to themobile terminal 100. The audio input can be processed in various mannersaccording to a function being executed in the mobile terminal 100. Ifdesired, the microphone 122 may include assorted noise removingalgorithms to remove unwanted noise generated in the course of receivingthe external audio.

The user input unit 123 is a component that permits input by a user.Such user input may enable the controller 180 to control operation ofthe mobile terminal 100. The user input unit 123 may include one or moreof a mechanical input element (for example, a key, a button located on afront and/or rear surface or a side surface of the mobile terminal 100,a dome switch, a jog wheel, a jog switch, and the like), or atouch-sensitive input, among others. As one example, the touch-sensitiveinput may be a virtual key or a soft key, which is displayed on a touchscreen through software processing, or a touch key which is located onthe mobile terminal at a location that is other than the touch screen.On the other hand, the virtual key or the visual key may be displayed onthe touch screen in various shapes, for example, graphic, text, icon,video, or a combination thereof.

The sensing unit 140 is generally configured to sense one or more ofinternal information of the mobile terminal, surrounding environmentinformation of the mobile terminal, user information, or the like. Thecontroller 180 generally cooperates with the sending unit 140 to controloperation of the mobile terminal 100 or execute data processing, afunction or an operation associated with an application programinstalled in the mobile terminal based on the sensing provided by thesensing unit 140. The sensing unit 140 may be implemented using any of avariety of sensors, some of which will now be described in more detail.

The proximity sensor 141 may include a sensor to sense presence orabsence of an object approaching a surface, or an object located near asurface, by using an electromagnetic field, infrared rays, or the likewithout a mechanical contact. The proximity sensor 141 may be arrangedat an inner region of the mobile terminal covered by the touch screen,or near the touch screen.

The proximity sensor 141, for example, may include any of a transmissivetype photoelectric sensor, a direct reflective type photoelectricsensor, a mirror reflective type photoelectric sensor, a high-frequencyoscillation proximity sensor, a capacitance type proximity sensor, amagnetic type proximity sensor, an infrared rays proximity sensor, andthe like. When the touch screen is implemented as a capacitance type,the proximity sensor 141 can sense proximity of a pointer relative tothe touch screen by changes of an electromagnetic field, which isresponsive to an approach of an object with conductivity. In this case,the touch screen (touch sensor) may also be categorized as a proximitysensor.

The term “proximity touch” will often be referred to herein to denotethe scenario in which a pointer is positioned to be proximate to thetouch screen without contacting the touch screen. The term “contacttouch” will often be referred to herein to denote the scenario in whicha pointer makes physical contact with the touch screen. For the positioncorresponding to the proximity touch of the pointer relative to thetouch screen, such position will correspond to a position where thepointer is perpendicular to the touch screen. The proximity sensor 141may sense proximity touch, and proximity touch patterns (for example,distance, direction, speed, time, position, moving status, and thelike).

In general, controller 180 processes data corresponding to proximitytouches and proximity touch patterns sensed by the proximity sensor 141,and cause output of visual information on the touch screen. In addition,the controller 180 can control the mobile terminal 100 to executedifferent operations or process different data according to whether atouch with respect to a point on the touch screen is either a proximitytouch or a contact touch.

A touch sensor can sense a touch applied to the touch screen, such asdisplay unit 151, using any of a variety of touch methods. Examples ofsuch touch methods include a resistive type, a capacitive type, aninfrared type, and a magnetic field type, among others.

As one example, the touch sensor may be configured to convert changes ofpressure applied to a specific part of the display unit 151, or convertcapacitance occurring at a specific part of the display unit 151, intoelectric input signals. The touch sensor may also be configured to sensenot only a touched position and a touched area, but also touch pressureand/or touch capacitance. A touch object is generally used to apply atouch input to the touch sensor. Examples of typical touch objectsinclude a finger, a touch pen, a stylus pen, a pointer, or the like.

When a touch input is sensed by a touch sensor, corresponding signalsmay be transmitted to a touch controller. The touch controller mayprocess the received signals, and then transmit corresponding data tothe controller 180. Accordingly, the controller 180 may sense whichregion of the display unit 151 has been touched. Here, the touchcontroller may be a component separate from the controller 180, thecontroller 180, and combinations thereof.

In some embodiments, the controller 180 may execute the same ordifferent controls according to a type of touch object that touches thetouch screen or a touch key provided in addition to the touch screen.Whether to execute the same or different control according to the objectwhich provides a touch input may be decided based on a current operatingstate of the mobile terminal 100 or a currently executed applicationprogram, for example.

The touch sensor and the proximity sensor may be implementedindividually, or in combination, to sense various types of touches. Suchtouches includes a short (or tap) touch, a long touch, a multi-touch, adrag touch, a flick touch, a pinch-in touch, a pinch-out touch, a swipetouch, a hovering touch, and the like.

If desired, an ultrasonic sensor may be implemented to recognizeposition information relating to a touch object using ultrasonic waves.The controller 180, for example, may calculate a position of a wavegeneration source based on information sensed by an illumination sensorand a plurality of ultrasonic sensors. Since light is much faster thanultrasonic waves, the time for which the light reaches the opticalsensor is much shorter than the time for which the ultrasonic wavereaches the ultrasonic sensor. The position of the wave generationsource may be calculated using this fact. For instance, the position ofthe wave generation source may be calculated using the time differencefrom the time that the ultrasonic wave reaches the sensor based on thelight as a reference signal.

The camera 121 typically includes at least one a camera sensor (CCD,CMOS etc.), a photo sensor (or image sensors), and a laser sensor.

Implementing the camera 121 with a laser sensor may allow detection of atouch of a physical object with respect to a 3D stereoscopic image. Thephoto sensor may be laminated on, or overlapped with, the displaydevice. The photo sensor may be configured to scan movement of thephysical object in proximity to the touch screen. In more detail, thephoto sensor may include photo diodes and transistors at rows andcolumns to scan content received at the photo sensor using an electricalsignal which changes according to the quantity of applied light. Namely,the photo sensor may calculate the coordinates of the physical objectaccording to variation of light to thus obtain position information ofthe physical object.

The display unit 151 is generally configured to output informationprocessed in the mobile terminal 100. For example, the display unit 151may display execution screen information of an application programexecuting at the mobile terminal 100 or user interface (UI) and graphicuser interface (GUI) information in response to the execution screeninformation.

In some embodiments, the display unit 151 may be implemented as astereoscopic display unit for displaying stereoscopic images. A typicalstereoscopic display unit may employ a stereoscopic display scheme suchas a stereoscopic scheme (a glass scheme), an auto-stereoscopic scheme(glassless scheme), a projection scheme (holographic scheme), or thelike.

In general, a 3D stereoscopic image may include a left image (e.g., aleft eye image) and a right image (e.g., a right eye image). Accordingto how left and right images are combined into a 3D stereoscopic image,a 3D stereoscopic imaging method can be divided into a top-down methodin which left and right images are located up and down in a frame, anL-to-R (left-to-right or side by side) method in which left and rightimages are located left and right in a frame, a checker board method inwhich fragments of left and right images are located in a tile form, aninterlaced method in which left and right images are alternately locatedby columns or rows, and a time sequential (or frame by frame) method inwhich left and right images are alternately displayed on a time basis.

Also, as for a 3D thumbnail image, a left image thumbnail and a rightimage thumbnail can be generated from a left image and a right image ofan original image frame, respectively, and then combined to generate asingle 3D thumbnail image. In general, the term “thumbnail” may be usedto refer to a reduced image or a reduced still image. A generated leftimage thumbnail and right image thumbnail may be displayed with ahorizontal distance difference there between by a depth corresponding tothe disparity between the left image and the right image on the screen,thereby providing a stereoscopic space sense.

A left image and a right image required for implementing a 3Dstereoscopic image may be displayed on the stereoscopic display unitusing a stereoscopic processing unit. The stereoscopic processing unitcan receive the 3D image and extract the left image and the right image,or can receive the 2D image and change it into a left image and a rightimage.

The audio output module 152 is generally configured to output audiodata. Such audio data may be obtained from any of a number of differentsources, such that the audio data may be received from the wirelesscommunication unit 110 or may have been stored in the memory 170. Theaudio data may be output during modes such as a signal reception mode, acall mode, a record mode, a voice recognition mode, a broadcastreception mode, and the like. The audio output module 152 can provideaudible output related to a particular function (e.g., a call signalreception sound, a message reception sound, etc.) performed by themobile terminal 100. The audio output module 152 may also be implementedas a receiver, a speaker, a buzzer, or the like.

A haptic module 153 can be configured to generate various tactileeffects that a user feels, perceive, or otherwise experience. A typicalexample of a tactile effect generated by the haptic module 153 isvibration. The strength, pattern and the like of the vibration generatedby the haptic module 153 can be controlled by user selection or settingby the controller. For example, the haptic module 153 may outputdifferent vibrations in a combining manner or a sequential manner.

Besides vibration, the haptic module 153 can generate various othertactile effects, including an effect by stimulation such as a pinarrangement vertically moving to contact skin, a spray force or suctionforce of air through a jet orifice or a suction opening, a touch to theskin, a contact of an electrode, electrostatic force, an effect byreproducing the sense of cold and warmth using an element that canabsorb or generate heat, and the like.

The haptic module 153 can also be implemented to allow the user to feela tactile effect through a muscle sensation such as the user's fingersor arm, as well as transferring the tactile effect through directcontact. Two or more haptic modules 153 may be provided according to theparticular configuration of the mobile terminal 100.

An optical output module 154 can output a signal for indicating an eventgeneration using light of a light source. Examples of events generatedin the mobile terminal 100 may include message reception, call signalreception, a missed call, an alarm, a schedule notice, an emailreception, information reception through an application, and the like.

A signal output by the optical output module 154 may be implemented insuch a manner that the mobile terminal emits monochromatic light orlight with a plurality of colors. The signal output may be terminated asthe mobile terminal senses that a user has checked the generated event,for example.

The interface unit 160 serves as an interface for external devices to beconnected with the mobile terminal 100. For example, the interface unit160 can receive data transmitted from an external device, receive powerto transfer to elements and components within the mobile terminal 100,or transmit internal data of the mobile terminal 100 to such externaldevice. The interface unit 160 may include wired or wireless headsetports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,or the like.

The identification module may be a chip that stores various informationfor authenticating authority of using the mobile terminal 100 and mayinclude a user identity module (UIM), a subscriber identity module(SIM), a universal subscriber identity module (USIM), and the like. Inaddition, the device having the identification module (also referred toherein as an “identifying device”) may take the form of a smart card.Accordingly, the identifying device can be connected with the terminal100 via the interface unit 160.

When the mobile terminal 100 is connected with an external cradle, theinterface unit 160 can serve as a passage to allow power from the cradleto be supplied to the mobile terminal 100 or may serve as a passage toallow various command signals input by the user from the cradle to betransferred to the mobile terminal there through. Various commandsignals or power input from the cradle may operate as signals forrecognizing that the mobile terminal is properly mounted on the cradle.

The memory 170 can store programs to support operations of thecontroller 180 and store input/output data (for example, phonebook,messages, still images, videos, etc.). The memory 170 may store datarelated to various patterns of vibrations and audio which are output inresponse to touch inputs on the touch screen.

The memory 170 may include one or more types of storage mediumsincluding a Flash memory, a hard disk, a solid state disk, a silicondisk, a multimedia card micro type, a card-type memory (e.g., SD or DXmemory, etc.), a Random Access Memory (RAM), a Static Random AccessMemory (SRAM), a Read-Only Memory (ROM), an Electrically ErasableProgrammable Read-Only Memory (EEPROM), a Programmable Read-Only memory(PROM), a magnetic memory, a magnetic disk, an optical disk, and thelike. The mobile terminal 100 may also be operated in relation to anetwork storage device that performs the storage function of the memory170 over a network, such as the Internet.

The controller 180 may typically control the general operations of themobile terminal 100. For example, the controller 180 may set or releasea lock state for restricting a user from inputting a control commandwith respect to applications when a status of the mobile terminal meetsa preset condition.

The controller 180 can also perform the controlling and processingassociated with voice calls, data communications, video calls, and thelike, or perform pattern recognition processing to recognize ahandwriting input or a picture drawing input performed on the touchscreen as characters or images, respectively. In addition, thecontroller 180 can control one or a combination of those components inorder to implement various exemplary embodiments disclosed herein.

The power supply unit 190 receives external power or provide internalpower and supply the appropriate power required for operating respectiveelements and components included in the mobile terminal 100. The powersupply unit 190 may include a battery, which is typically rechargeableor be detachably coupled to the terminal body for charging.

The power supply unit 190 may include a connection port. The connectionport may be configured as one example of the interface unit 160 to whichan external charger for supplying power to recharge the battery iselectrically connected.

As another example, the power supply unit 190 may be configured torecharge the battery in a wireless manner without use of the connectionport. In this example, the power supply unit 190 can receive power,transferred from an external wireless power transmitter, using at leastone of an inductive coupling method which is based on magnetic inductionor a magnetic resonance coupling method which is based onelectromagnetic resonance.

Various embodiments described herein may be implemented in acomputer-readable medium, a machine-readable medium, or similar mediumusing, for example, software, hardware, or any combination thereof.

FIG. 2 is a perspective view illustrating one example of a glass-typemobile terminal 400 according to another exemplary embodiment.

The glass-type mobile terminal 400 can be wearable on a head of a humanbody and provided with a frame (case, housing, etc.) therefor. The framemay be made of a flexible material to be easily worn. The frame ofmobile terminal 400 is shown having a first frame 401 and a second frame402, which can be made of the same or different materials. In general,mobile terminal 400 may be configured to include features that are thesame or similar to that of mobile terminal 100 of FIG. 1.

The frame may be supported on the head and defines a space for mountingvarious components. As illustrated, electronic components, such as acontrol module 480, an audio output module 452, and the like, may bemounted to the frame part. Also, a lens 403 for covering either or bothof the left and right eyes may be detachably coupled to the frame part.

The control module 480 controls various electronic components disposedin the mobile terminal 400. The control module 480 may be understood asa component corresponding to the aforementioned controller 180. FIG. 4illustrates that the control module 480 is installed in the frame parton one side of the head, but other locations are possible.

The display unit 451 may be implemented as a head mounted display (HMD).The HMD refers to display techniques by which a display is mounted to ahead to show an image directly in front of a user's eyes. In order toprovide an image directly in front of the user's eyes when the userwears the glass-type mobile terminal 400, the display unit 451 may belocated to correspond to either or both of the left and right eyes. FIG.4 illustrates that the display unit 451 is located on a portioncorresponding to the right eye to output an image viewable by the user'sright eye.

The display unit 451 may project an image into the user's eye using aprism. Also, the prism may be formed from optically transparent materialsuch that the user can view both the projected image and a generalvisual field (a range that the user views through the eyes) in front ofthe user.

In such a manner, the image output through the display unit 451 may beviewed while overlapping with the general visual field. The mobileterminal 400 may provide an augmented reality (AR) by overlaying avirtual image on a realistic image or background using the display.

The camera 421 may be located adjacent to either or both of the left andright eyes to capture an image. Since the camera 421 is located adjacentto the eye, the camera 421 can acquire a scene that the user iscurrently viewing. The camera 421 may be positioned at most any locationof the mobile terminal. In some embodiments, multiple cameras 421 may beutilized. Such multiple cameras 421 may be used to acquire astereoscopic image.

The glass-type mobile terminal 400 may include user input units 423 aand 423 b, which can each be manipulated by the user to provide aninput. The user input units 423 a and 423 b may employ techniques whichpermit input via a tactile input. Typical tactile inputs include atouch, push, or the like. The user input units 423 a and 423 b are shownoperable in a pushing manner and a touching manner as they are locatedon the frame part and the control module 480, respectively.

If desired, mobile terminal 400 may include a microphone which processesinput sound into electric audio data, and an audio output module 452 foroutputting audio. The audio output module 452 may be configured toproduce audio in a general audio output manner or an osteoconductivemanner. When the audio output module 452 is implemented in theosteoconductive manner, the audio output module 452 may be closelyadhered to the head when the user wears the mobile terminal 400 andvibrate the user's skull to transfer sounds.

A communication system which is operable with the variously describedmobile terminals will now be described in more detail. Such acommunication system may be configured to utilize any of a variety ofdifferent air interfaces and/or physical layers. Examples of such airinterfaces utilized by the communication system include FrequencyDivision Multiple Access (FDMA), Time Division Multiple Access (TDMA),Code Division Multiple Access (CDMA), Universal MobileTelecommunications System (UMTS) (including, Long Term Evolution (LTE),LTE-A (Long Term Evolution-Advanced)), Global System for MobileCommunications (GSM), and the like.

By way of a non-limiting example only, further description will relateto a CDMA communication system, but such teachings apply equally toother system types including a CDMA wireless communication system aswell as OFDM (Orthogonal Frequency Division Multiplexing) wirelesscommunication system. A CDMA wireless communication system generallyincludes one or more mobile terminals (MT or User Equipment, UE) 100,one or more base stations (BSs, NodeB, or evolved NodeB), one or morebase station controllers (BSCs), and a mobile switching center (MSC).The MSC is configured to interface with a conventional Public SwitchedTelephone Network (PSTN) and the BSCs. The BSCs are coupled to the basestations via backhaul lines. The backhaul lines may be configured inaccordance with any of several known interfaces including, for example,E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. Hence, theplurality of BSCs can be included in the CDMA wireless communicationsystem.

Each base station may include one or more sectors, each sector having anomni-directional antenna or an antenna pointed in a particular directionradially away from the base station. Alternatively, each sector mayinclude two or more different antennas. Each base station may beconfigured to support a plurality of frequency assignments, with eachfrequency assignment having a particular spectrum (e.g., 1.25 MHz, 5MHz, etc.).

The intersection of sector and frequency assignment may be referred toas a CDMA channel. The base stations may also be referred to as BaseStation Transceiver Subsystems (BTSs). In some cases, the term “basestation” may be used to refer collectively to a BSC, and one or morebase stations. The base stations may also be denoted as “cell sites.”Alternatively, individual sectors of a given base station may bereferred to as cell sites.

A broadcasting transmitter (BT) transmits a broadcast signal to themobile terminals 100 operating within the system. The broadcastreceiving module 111 of FIG. 1A is typically configured inside themobile terminal 100 to receive broadcast signals transmitted by the BT.

Global Positioning System (GPS) satellites for locating the position ofthe mobile terminal 100, for example, may cooperate with the CDMAwireless communication system. Useful position information may beobtained with greater or fewer satellites than two satellites. It is tobe appreciated that other types of position detection technology, (e.g.,location technology that may be used in addition to or instead of GPSlocation technology) may alternatively be implemented. If desired, atleast one of the GPS satellites may alternatively or additionally beconfigured to provide satellite DMB transmissions.

The location information module 115 is generally configured to detect,calculate, or otherwise identify a position of the mobile terminal. Asan example, the location information module 115 may include a GlobalPosition System (GPS) module, a Wi-Fi module, or both. If desired, thelocation information module 115 may alternatively or additionallyfunction with any of the other modules of the wireless communicationunit 110 to obtain data related to the position of the mobile terminal.

A typical GPS module 115 can measure an accurate time and distance fromthree or more satellites, and accurately calculate a current location ofthe mobile terminal according to trigonometry based on the measured timeand distances. A method of acquiring distance and time information fromthree satellites and performing error correction with a single satellitemay be used. In particular, the GPS module may acquire an accurate timetogether with three-dimensional speed information as well as thelocation of the latitude, longitude and altitude values from thelocation information received from the satellites.

Furthermore, the GPS module can acquire speed information in real timeto calculate a current position. Sometimes, accuracy of a measuredposition may be compromised when the mobile terminal is located in ablind spot of satellite signals, such as being located in an indoorspace. In order to minimize the effect of such blind spots, analternative or supplemental location technique, such as Wi-FiPositioning System (WPS), may be utilized.

The Wi-Fi positioning system (WPS) refers to a location determinationtechnology based on a wireless local area network (WLAN) using Wi-Fi asa technology for tracking the location of the mobile terminal 100. Thistechnology typically includes the use of a Wi-Fi module in the mobileterminal 100 and a wireless access point for communicating with theWi-Fi module.

The Wi-Fi positioning system may include a Wi-Fi location determinationserver, a mobile terminal, a wireless access point (AP) connected to themobile terminal, and a database stored with wireless AP information.

The mobile terminal connected to the wireless AP may transmit a locationinformation request message to the Wi-Fi location determination server.The Wi-Fi location determination server extracts the information of thewireless AP connected to the mobile terminal 100, based on the locationinformation request message (or signal) of the mobile terminal 100. Theinformation of the wireless AP may be transmitted to the Wi-Fi locationdetermination server through the mobile terminal 100, or may betransmitted to the Wi-Fi location determination server from the wirelessAP.

The information of the wireless AP extracted based on the locationinformation request message of the mobile terminal 100 may include oneor more of media access control (MAC) address, service setidentification (SSID), received signal strength indicator (RSSI),reference signal received Power (RSRP), reference signal receivedquality (RSRQ), channel information, privacy, network type, signalstrength, noise strength, and the like.

The Wi-Fi location determination server may receive the information ofthe wireless AP connected to the mobile terminal 100 as described above,and may extract wireless AP information corresponding to the wireless APconnected to the mobile terminal from the pre-established database. Theinformation of any wireless APs stored in the database may beinformation such as MAC address, SSID, RSSI, channel information,privacy, network type, latitude and longitude coordinate, building atwhich the wireless AP is located, floor number, detailed indoor locationinformation (GPS coordinate available), AP owner's address, phonenumber, and the like. In order to remove wireless APs provided using amobile AP or an illegal MAC address during a location determiningprocess, the Wi-Fi location determination server may extract only apredetermined number of wireless AP information in order of high RSSI.

Then, the Wi-Fi location determination server may extract (analyze)location information of the mobile terminal 100 using at least onewireless AP information extracted from the database.

A method for extracting (analyzing) location information of the mobileterminal 100 may include a Cell-ID method, a fingerprint method, atrigonometry method, a landmark method, and the like.

The Cell-ID method is used to determine a position of a wireless APhaving the largest signal strength, among peripheral wireless APinformation collected by a mobile terminal, as a position of the mobileterminal. The Cell-ID method is an implementation that is minimallycomplex, does not require additional costs, and location information canbe rapidly acquired. However, in the Cell-ID method, the precision ofpositioning may fall below a desired threshold when the installationdensity of wireless APs is low.

The fingerprint method is used to collect signal strength information byselecting a reference position from a service area, and to track aposition of a mobile terminal using the signal strength informationtransmitted from the mobile terminal based on the collected information.In order to use the fingerprint method, it is common for thecharacteristics of radio signals to be pre-stored in the form of adatabase.

The trigonometry method is used to calculate a position of a mobileterminal based on a distance between coordinates of at least threewireless APs and the mobile terminal. In order to measure the distancebetween the mobile terminal and the wireless APs, signal strength may beconverted into distance information, Time of Arrival (ToA), TimeDifference of Arrival (TDoA), Angle of Arrival (AoA), or the like may betaken for transmitted wireless signals.

The landmark method is used to measure a position of a mobile terminalusing a known landmark transmitter.

In addition to these position location methods, various algorithms maybe used to extract (analyze) location information of a mobile terminal.Such extracted location information may be transmitted to the mobileterminal 100 through the Wi-Fi location determination server, therebyacquiring location information of the mobile terminal 100.

The mobile terminal 100 can acquire location information by beingconnected to at least one wireless AP. The number of wireless APsrequired to acquire location information of the mobile terminal 100 maybe variously changed according to a wireless communication environmentwithin which the mobile terminal 100 is positioned.

As previously described with regard to FIG. 1, the mobile terminal maybe configured to include short-range communication techniques such asBluetooth™, Radio Frequency Identification (RFID), Infrared DataAssociation (IrDA), UltraWideband (UWB), ZigBee, Near FieldCommunication (NFC), Wireless USB (Wireless Universal Serial Bus), andthe like.

A typical NFC module provided at the mobile terminal supportsshort-range wireless communication, which is a non-contactable type ofcommunication between mobile terminals and generally occurs within about10 cm. The NFC module may operate in one of a card mode, a reader mode,or a P2P mode. The mobile terminal 100 may further include a securitymodule for storing card information, in order to operate the NFC modulein a card mode. The security module may be a physical medium such asUniversal Integrated Circuit Card (UICC) (e.g., a SubscriberIdentification Module (SIM) or Universal SIM (USIM)), a secure micro SDand a sticker, or a logical medium (e.g., embedded Secure Element (SE))embedded in the mobile terminal. Single Wire Protocol (SWP)-based dataexchange may be performed between the NFC module and the securitymodule.

In a case where the NFC module operates in a card mode, the mobileterminal may transmit card information on a general IC card to theoutside. More specifically, if a mobile terminal having card informationon a payment card (e. g, a credit card or a bus card) approaches a cardreader, a short-range mobile payment may be executed. As anotherexample, if a mobile terminal which stores card information on anentrance card approaches an entrance card reader, an entrance approvalprocedure may start. A card such as a credit card, a traffic card, or anentrance card may be included in the security module in the form ofapplet, and the security module may store card information on the cardmounted therein. Card information for a payment card may include any ofa card number, a remaining amount and usage history, and the like. Cardinformation of an entrance card may include any of a user's name, auser's number (e.g., undergraduate number or staff number), an entrancehistory, and the like.

When the NFC module operates in a reader mode, the mobile terminal canread data from an external tag. The data received from the external tagby the mobile terminal may be coded into the NFC Data Exchange Formatdefined by the NFC Forum. The NFC Forum generally defines four recordtypes. More specifically, the NFC Forum defines four Record TypeDefinitions (RTDs) such as smart poster, text, Uniform ResourceIdentifier (URI), and general control. If the data received from theexternal tag is a smart poster type, the controller may execute abrowser (e.g., Internet browser). If the data received from the externaltag is a text type, the controller may execute a text viewer. If thedata received from the external tag is a URI type, the controller mayexecute a browser or originate a call. If the data received from theexternal tag is a general control type, the controller may execute aproper operation according to control content.

In some cases in which the NFC module operates in a P2P (Peer-to-Peer)mode, the mobile terminal can execute P2P communication with anothermobile terminal. In this case, Logical Link Control Protocol (LLCP) maybe applied to the P2P communication. For P2P communication, connectionmay be generated between the mobile terminal and another mobileterminal. This connection may be categorized as a connectionless modewhich ends after one packet is switched, and a connection-oriented modein which packets are switched consecutively. For a typical P2Pcommunication, data such as an electronic type name card, addressinformation, a digital photo and a URL, a setup parameter for Bluetoothconnection, Wi-Fi connection, etc. may be switched. The P2P mode can beeffectively utilized in switching data of a small capacity, because anavailable distance for NFC communication is relatively short.

Hereinafter, embodiments related to a method of displaying a realtraveling path of a robot cleaner using an augmented reality (AR) image,which may be implemented in the mobile terminal 100 having theabove-described configuration, will be described with reference to theaccompanying drawings. However, those skilled in the art will appreciatethat the present invention may be implemented in the form of a varietyof formats without departing from the spirit or scope of the invention.

FIG. 3 is a diagram illustrating the configuration of an augmentedreality (AR) system according to an embodiment of the present invention.

The AR system according to the embodiment of the present invention mayinclude a mobile terminal 100, an electronic device 200 and a server300.

The operation mode of the mobile terminal 100 may include an AR mode.The AR mode may refer to a mode for superimposing a virtual image on areal image to display one image. Here, the virtual image may be relatedto the real image and may include information on the real image.

The mobile terminal 100 may enter the AR mode by executing anapplication installed therein. Upon entering the AR mode, the mobileterminal 100 may automatically display a preview image acquired throughthe camera 121 on the display unit 151.

The mobile terminal 100 may identify the electronic device 200 throughthe preview image and provide information on the identified electronicdevice 200.

In one embodiment, the controller 180 of the mobile terminal 100 mayidentify the electronic device 200 using an image recognition scheme.The controller 180 of the mobile terminal 100 may identify theelectronic device 200 based on the preview image of the electronicdevice 200 acquired through the camera 121.

The preview image may be an image which is displayed before beingcaptured through the camera 121. The controller 180 may acquire theexternal image of the electronic device 200 included in the previewimage and compare the acquired external image of the electronic device200 with the external images of the electronic devices stored in thememory 170 of the mobile terminal 100.

When the acquired external image of the electronic device 200 is storedin the memory 170 as the result of comparison, the controller 180 mayextract information on the stored electronic device. The information onthe electronic device may be at least one of the name of the electronicdevice, the model name of the electronic device, information on partsprovided in the electronic device, the image information of parts, andthe position information of the internal parts of the electronic device.

When the acquired external image of the electronic device 200 is notstored in the memory 170 as the result of comparison, the controller 180may access the server 300 including a database of the electronic devicethrough the wireless Internet module 113. The server 300 may identifythe electronic device 200 through comparison between the external imageof the electronic device 200 and the stored external images of theelectronic devices from the database.

The controller 180 may receive the information on the electronic device200 identified by the server 300.

In another embodiment, the controller 180 may identify the electronicdevice 200 based on the identifier attached to the electronic device200. The identifier may be any one of a barcode, a QR code and an RFIDand may include information for identifying the electronic device 200.

The controller 180 may recognize the identifier included in the previewimage acquired through the camera 121 and identify the electronic device200.

In another embodiment, the controller 180 may identify the electronicdevice 200 based on the position information of the mobile terminal 100.Specifically, the controller 180 may identify the electronic device 200displayed in the preview image through comparison between the positioninformation of the mobile terminal 100 and the position information ofthe electronic device 200.

Although GPS information is preferably used as position information, itmay be difficult to acquire GPS information indoors. Therefore, thecontroller 180 may use a method of providing a Wi-Fi position service(WPS) using the wireless Internet, a method of using Bluetooth, a methodof using RFID, etc.

The short-range communication module 114 of the mobile terminal 100 mayperform short-range wireless communication with the electronic device200. To this end, the electronic device 200 may also include ashort-range communication module.

The electronic device 200 may be any one of an air conditioner, arefrigerator, a washing machine, a TV, a microwave oven, a cleaner andan airport robot. However, the present invention is not limited theretoand the electronic device 200 may include all types of devices forincreasing user convenience.

The server 300 may exchange information with the mobile terminal 100through the Internet.

In some cases, the server 300 may be connected to the electronic device200 through wireless communication. In this case, the electronic device200 may also include a wireless Internet module.

According to one embodiment, the electronic device 100 may be a robotcleaner 10. In this case, the AR system according to the embodiment ofthe present invention may include a robot cleaner 10 for performingcleaning while automatically traveling and a mobile terminal 100 forentering an AR mode and displaying the traveling path of the robotcleaner 10 using an AR image.

Hereinafter, the configuration of the robot cleaner 10 will be describedfirst.

FIGS. 4A to 4 c are diagrams illustrating the configuration of a robotcleaner according to an embodiment of the present invention.

Specifically, FIG. 4A is a perspective view showing the robot cleaner,FIG. 4B is a perspective view showing the internal configuration of therobot cleaner, and FIG. 4C is a perspective view showing the lower sideof the robot cleaner.

The robot cleaner 10 according to the embodiment of the presentinvention may include a main body 11, a suction device 12, a suctionnozzle 13, a dust collector 14, driving wheels 15 a, motors 15 b and anauxiliary wheel 15 c.

The main body 11 forms the appearance of the robot cleaner 10.

The main body 11 may have a cylindrical shape and have a height that isless than a diameter thereof, that is, a flat cylindrical shape. In thiscase, the suction device 12, the suction nozzle 13, and the dustcollector 14 communicating with the suction nozzle 13 may be provided inthe main body 11.

The suction device 12 generates air suction force.

The dust collector 14 may be provided at the rear side of the suctiondevice 12. In this case, the suction device 12 may include a motor (notshown) obliquely provided between a battery 560 and the dust collector14 and electrically connected to the battery 560 and a fan (not shown)connected to the rotation shaft of the motor to forcibly flow air.

The suction nozzle 13 sucks in dust located on the floor by driving thesuction device 12. Specifically, the suction nozzle 13 is exposed to thelower side of the main body 11 through an opening (not shown) formed inthe bottom of the main body 11, thereby being in contact with the indoorfloor. In this case, the suction nozzle 13 may suck in foreign materialson the floor along with air.

The dust collector 14 collects foreign materials in air sucked in by thesuction nozzle 13. To this end, the suction nozzle 13 is provided at thelower side of the dust collector 14.

A sensor (not shown) for sensing a distance from an indoor wall or anobstacle, a bumper (not shown) for absorbing impact upon collision, andthe driving wheel 15 a for moving the robot cleaner 10 may be providedon the outer circumferential surface of the main body 11.

The driving wheels 15 a may be provided at the lower side of the mainbody 11. Specifically, the driving wheels 15 a may be provided at bothsides, that is, the left and right sides, of the lower side of the mainbody.

The driving wheels 15 a are configured to rotate by the motors 15 bcontrolled by the control unit 570. To this end, the motors 15 b may beprovided to correspond to the driving wheels 15 a, that is, may beprovided at both sides, the left and right sides, of the lower side ofthe main body. In this case, the motors 15 b provided at the left andright sides may independently operate. Accordingly, the robot cleaner 10may move forward or backward or turn left or right. Therefore, the robotcleaner 10 may perform indoor cleaning, while driving the motors 15 b toautonomously change a direction thereof.

At least one auxiliary wheel 15 c may be provided on the bottom of themain body 11. The auxiliary wheel 15 c may minimize friction between therobot cleaner 10 and the floor and guide movement of the robot cleaner10.

FIG. 5 is a block diagram showing the configuration of a robot cleaneraccording to an embodiment of the present invention.

The robot cleaner 10 according to the embodiment of the presentinvention may include a communication unit 510, a camera 520, an inputunit 530, a display unit 540, a storage unit 550, a battery 560 and acontrol unit 570.

The communication unit 510 may perform communication with the mobileterminal 100. The robot cleaner 10 may be controlled by the mobileterminal 100 or transmit or receive data and/or control commands to orfrom the mobile terminal 100.

Communication performed between the communication unit 510 and themobile terminal 100 may include a variety of wired or wirelesscommunication such as low-power Bluetooth communication, Bluetoothcommunication, RFID communication, infrared communication, wireless LANcommunication, wireless communication, etc. To this end, thecommunication unit 510 may include a communication module capable ofperforming wired/wireless communication.

The camera 520 generates image information of a predetermined region.Here, the predetermined region may include the robot cleaner 10 and theperipheral region of the robot cleaner 10. The image information may bea still image or a moving image. To this end, the camera 520 may capturethe robot cleaner 10 and the peripheral region of the robot cleaner 10and generate a still image or a moving image.

In one embodiment, in order to acquire image information of variousregions, a plurality of cameras 520 may be provided in the robot cleaner10. In this case, at least one of the plurality of cameras 520 may beprovided at the upper side of the robot cleaner 10 and at least one ofthe cameras may be provided at the circumference of the side surface ofthe robot cleaner 10.

The input unit 530 may receive a command related to operation or controlof the robot cleaner 100.

To this end, the input unit 530 may include a plurality of buttonsconfigured in the form of a push key or a touch key. The plurality ofbuttons may include at least one of a setting button for setting an itemrelated to operation or control of the robot cleaner 10, a confirmbutton for confirming an input item, a delete button for deleting a setitem, a start button for starting cleaning and a stop button forstopping cleaning.

Here, the command related to operation or control of the robot cleaner10 may be a cleaning scheduling command, a cleaning start command or acleaning end command. Alternatively, the command related to operation orcontrol of the robot cleaner 10 may be a command related to setting ofthe cleaning pattern (or traveling pattern) of the robot cleaner 10.Here, the cleaning pattern may include at least one of a zigzag pattern,a random pattern, a per-cell pattern (a pattern for performing cleaningin a specific cell and then performing cleaning in a next cell) and auser setting pattern.

The display unit 540 may display information related to operation orcontrol of the robot cleaner 10.

Specifically, operation information of the robot cleaner 10, imageinformation acquired by the camera 520, cleaning map information of acleaning region, time information of the robot cleaner 10 may bedisplayed on the display unit 540. Here, the cleaning region may includea target-of-cleaning region, a non-target-of-cleaning region, a cleanedregion, an uncleaned region, etc. Hereinafter, in the present invention,the target-of-cleaning region means a region in which cleaning isperformed and the non-target-of-cleaning region means a region in whichcleaning is not performed. In addition, the cleaned region means aregion in which cleaning has already been performed and the uncleanedregion means a region in which cleaning is not yet performed. Inaddition, the time information of the robot cleaner 10 may include atleast one of a cleaning start time, a cleaning end time, a requiredcleaning time, a residual time, an estimated time and a current time ofthe robot cleaner 10.

The storage unit 550 may store data.

Specifically, the target-of-cleaning region, the non-target-of-cleaningregion, the cleaned region and the uncleaned region may bedistinguishably stored in the storage unit 550 in association with thecleaning process. In this case, data on the target-of-cleaning region,the non-target-of-cleaning region, the cleaned region and the uncleanedregion may be periodically updated in correspondence with operation ofthe robot cleaner 10.

The battery 560 may supply power to the robot cleaner 10 and thecomponents thereof. The battery 560 may be a built-in battery or areplaceable battery. In this case, the battery 560 is rechargeable andmay be detachably coupled to the main body for charging.

When the robot cleaner 10 is connected to a docking station, the battery560 may be charged by power received from the docking station. Accordingto one embodiment, when cleaning operation is finished, the robotcleaner 10 may be automatically switched to a docking mode and moved andconnected to the docking station.

The control unit 570 may be connected to the various components includedin the robot cleaner 10 to control overall operation of the robotcleaner 10. Such a control unit 570 may be provided in the main body 11at a predetermined position.

The robot cleaner 10 according to the embodiment of the presentinvention may further include a sound output unit (not shown). The soundoutput unit (not shown) may audibly output operation information of therobot cleaner 10. In addition, the sound output unit (not shown) mayaudibly output information received from an external device. Forexample, when the robot cleaner 10 receives control information or textinformation from the mobile terminal 100, the robot cleaner 10 mayconvert the control information or the text information to sound andoutput sound. According to one embodiment, the sound output unit (notshown) may be a speaker.

FIGS. 6A and 6B are diagrams illustrating the case where a mobileterminal according to an embodiment of the present invention operates inan AR mode.

The mobile terminal 100 according to the embodiment of the presentinvention may activate the AR mode. The AR mode may be a mode in which aconnectable electronic device 200 is recognized and the recognizedelectronic device 200 or information related thereto is displayed on thescreen of the mobile terminal 100 using an AR image. Here, theelectronic device 200 may be a robot cleaner 10.

The mobile terminal 100 may recognize the connectable robot cleaner 10.Specifically, the mobile terminal 100 may recognize the robot cleaner 10located in a region where the mobile terminal has entered or the robotcleaner 10 located within a predetermined distance from the mobileterminal 100. In addition, the mobile terminal 100 may recognize therobot cleaner 10 set by the user.

Upon recognizing the connectable robot cleaner 10, the mobile terminal100 may output an alarm indicating that the robot cleaner 10 isrecognized to the user. Such an alarm may be displayed on the screen inthe form of at least one of an image and text or may be output in theform of at least one of an image and text or at least one of vibrationand sound.

In order to set the connectable robot cleaner 10, the mobile terminal100 may pair with the robot cleaner 10. Specifically, unique informationof the robot cleaner 10 paired with the mobile terminal 100 may be inputto the mobile terminal 100. For example, the unique information may be aserial number generated upon manufacturing the robot cleaner 10.However, the present invention is not limited thereto and the uniqueinformation may be various types of information capable of identifyingthe robot cleaner 100, such as an IP address allocated to the robotcleaner 10, a quick response (QR) code, etc. Pairing between the mobileterminal 100 and the robot cleaner 10 may be performed only upon initialconnection or whenever performing connection, according to embodiment.

In the AR mode, the mobile terminal 100 may perform communication withthe robot cleaner 10. Communication performed between the mobileterminal 100 and the robot cleaner 10 may include various wirelesscommunication methods such as Bluetooth low energy (BLE) communication,NFC, Bluetooth, infrared data association (IrDA) and wireless LANcommunication. According to an embodiment, the mobile terminal 100 mayperform BLE communication with the robot cleaner 10.

The mobile terminal 100 may start the AR mode according to variousembodiments. In one embodiment, the mobile terminal 100 may start the ARmode if the robot cleaner 10 is included in the screen captured by thecamera included in the mobile terminal 100. In another embodiment, themobile terminal 100 may start the AR mode when entering a region wherethe robot cleaner 10 is located or accessing the robot cleaner 10 withina predetermined distance from the robot cleaner 10 to recognize thepaired robot cleaner 10. In another embodiment, the mobile terminal 100may start the AR mode upon receiving a QR code attached to the robotcleaner 10.

According to an embodiment, when the robot cleaner connected to thedocking station is included in the screen 610 captured by the mobileterminal 100 in a state in which the robot cleaner 10 is in a dockingmode, the activated AR mode may be continuously maintained even when therobot cleaner 10 is not displayed on the screen of the mobile terminal100.

Meanwhile, if the AR mode starts, the mobile terminal 100 may display amessage for inquiring whether an AR image is displayed on the screen.

Referring to FIG. 6A, the robot cleaner 10 in the docked state isincluded in the screen 610 captured by the camera 121 provided in themobile terminal 100. The robot cleaner 10 pairs with the mobile terminal100. Accordingly, the mobile terminal 100 recognizes a connectable robotcleaner 100 and starts an AR mode.

In this case, although the robot cleaner 10 disappears on the screen ofthe mobile terminal 100 after the AR mode is activated, the AR mode iscontinuously maintained.

FIG. 6B shows a message 620 displayed on the screen of the mobileterminal 100 when the AR mode is activated. Referring to FIG. 6B, amessage 620 for inquiring whether the traveling path of the robotcleaner 10 is checked using an AR image is displayed on the screen 610of the mobile terminal 100.

FIGS. 7A and 7B are diagrams showing a cleaning map displayed by ageneral robot cleaner.

The robot cleaner may display a cleaning map. To this end, the robotcleaner may generate a cleaning map based on the traveling path. In thecleaning map, information on at least one of a target-of-cleaningregion, a non-target-of cleaning region, a cleaned region and anuncleaned region may be displayed. In this case, the regions may bedistinguishably displayed. For example, the regions may be displayed bylines, or figures or faces or may be displayed in different shapes orcolors.

Specifically, the robot cleaner may acquire the image information of thecleaning region and the peripheral region by the camera upon traveling,and map the target-of-cleaning region and the non-target-of-cleaningregion based on the acquired image information. For example, a region inwhich an obstacle is located may be mapped to the non-target-of-cleaningregion.

The robot cleaner may perform cleaning while traveling based on mappinginformation of the target-of-cleaning region and thenon-target-of-cleaning region. In this case, the robot cleaner mayacquire the image information of the cleaned region and the peripheralregion by the camera and map the cleaned region and the uncleaned regionbased on the acquired image information.

The robot cleaner may generate the cleaning map based on the mappinginformation. In this case, the mapping information and the cleaning mapmay be stored in the storage unit 550.

FIG. 7A shows an example of the cleaning map 700. The cleaning map 700includes a plurality of cells partitioning the target-of-cleaning regioninto a plurality of squares. In this case, a cleaning start point, acleaning end start and a traveling path 710 are displayed on thecleaning map 700. Here, the traveling path 710 is denoted by a solidline passing through the center of each cell.

The cleaning map 700 shown in FIG. 7A displays the traveling path 710 ofthe robot cleaner. However, according to the cleaning map 700, the usercannot check to which region of the real space the cleaning map 700corresponds.

FIG. 7B shows another example of the cleaning map 750. The cleaning mapis displayed to correspond to the real space. The cleaning map 750includes a plurality of circles, into which a target-of-cleaning regionis equally divided. In this case, cleaned regions X and uncleanedregions Y are displayed on the cleaning map 750. Specifically, thecleaned regions X are displayed as white circles and the uncleanedregions Y are displayed as black circles.

The cleaning map 750 shown in FIG. 7B is displayed to correspond to thereal space. Therefore, the user may check to which region of the realspace the cleaning map 750 corresponds. However, since only the wholeshape of the space is displayed on the cleaning map 750, the user cannotintuitively check the real space corresponding to the cleaning map 750.

FIG. 8 is a diagram illustrating an example in which a mobile terminalaccording to an embodiment of the present invention maps a real space toa virtual space.

The mobile terminal 100 according to the embodiment of the presentinvention may map a real space to a virtual space. Here, the real spacemay be a space where the robot cleaner 10 is actually located. Thevirtual space may be a space where the robot cleaner 10 is virtuallylocated, which is projected onto the screen of the mobile terminal 100by the camera.

In general, the robot cleaner 10 travels on the plane and does nottravel in a vertical direction. Accordingly, the coordinates of the realspace of the robot cleaner 10 may be represented by two-dimensionalcoordinates (x, y).

In contrast, the virtual space displayed on the screen of the mobileterminal 100 is a three-dimensional space. Accordingly, the coordinatesof the virtual space of the robot cleaner 10 may be represented bythree-dimensional coordinates (X, Y, Z).

In this case, the mobile terminal 100 may map the two-dimensionalcoordinates of the real space of the robot cleaner 10 to thethree-dimensional coordinates of the virtual space. Therefore, themobile terminal 100 may map the real region where the robot cleaner 10is located to the virtual region displayed on the screen of the mobileterminal 100.

The mobile terminal 100 may receive the position information of therobot cleaner 10 from the robot cleaner 10 and generate coordinates ofthe robot cleaner 10 in the real space. In this case, the robot cleaner10 may generate the position information of the robot cleaner 10 usingthe image information captured by the camera 520.

In addition, the mobile terminal 100 may generate the coordinates of thevirtual space based on the image information displayed on the screen ofthe mobile terminal 100.

Meanwhile, in some embodiments, mapping of the real space to the virtualspace may be performed by the robot cleaner 10 instead of the mobileterminal 100.

Referring to FIG. 8, two-dimensional coordinates (x, y) correspond tothree-dimensional coordinates (X, Y, Z). The mobile terminal 100 mayperform coordinate mapping to map the real space to the virtual spacedisplayed on the screen of the mobile terminal 100. Using such a method,the robot cleaner 10 may display the real cleaning region of thecleaning map on the screen of the mobile terminal 100.

FIGS. 9A to 9D are diagrams illustrating an example in which a mobileterminal according to an embodiment of the present invention displays atraveling path of a robot cleaner using an AR image.

The mobile terminal 100 according to the embodiment of the presentinvention may display the traveling path of the robot cleaner 10 usingthe AR image.

AR means that a virtual image is superimposed on a real image to displayone image. Specifically, AR may mean that a virtual image artificiallygenerated by a computer is displayed on an image of the real world. Inthe present invention, the virtual image included in AR is defined as anAR image.

The mobile terminal 100 may request information on the traveling pathfrom the robot cleaner 10, when a predetermined region of the real spaceis projected onto the screen captured by the built-in camera 121 in astate of activating the AR mode. In this case, the mobile terminal 100may receive information on the traveling path from the robot cleaner 10,map the traveling path of the real space to the traveling pat of thevirtual space, and display the traveling path using the AR image.

Here, the real space may include a region where cleaning is performed.

FIG. 9A shows the case where a real space is projected by a built-incamera. Referring to FIG. 9A, the mobile terminal 100 project the realspace using the built-in camera. Therefore, the predetermined region ofthe real space is included in the screen captured by the camera of themobile terminal 100. In this case, the mobile terminal 100 requests theinformation on the traveling path from the robot cleaner 10.

FIG. 9B shows an example of displaying the traveling path using the ARimage 910. The AR image 910 of the traveling path may be displayed incorrespondence with the traveling path. According to one embodiment, theAR image 910 may be indicated by a dotted line having a predeterminedtrajectory. In this case, the AR image 910 may include a dotted linecorresponding to the traveling path and cleaning start and end pointsdenoted by arrows and located at both ends of the line. The user cancheck the cleaned region and the order of cleaning.

The AR image 910 of the traveling path may be displayed to besuperimposed on the real image. In this case, the AR image 910 which isa virtual image may be displayed in the form of a stereoscopic image ora holographic image to be distinguished from the real image.

Referring to FIG. 9B, the virtual AR image 910 is displayed on the realimage displayed on the screen of the mobile terminal 100. In this case,the AR image includes a dotted line corresponding to the traveling pathfrom the cleaning start point to the cleaning end point.

FIG. 9C shows another example of displaying the traveling path using theAR image. The AR image of the traveling path may be displayed in theform of animation or a moving image. Specifically, the AR image mayinclude a cleaner icon 920 and a traveling path 930 and the cleaner icon920 may move along the traveling path 930. In this case, as the cleanericon 920 moves, the traveling path 930 of the cleaner may be changed inreal time.

The AR image of the traveling path may be displayed to be superimposedon the real image.

Referring to FIG. 9C, the AR image of the traveling path is displayed tobe superimposed on the real image displayed on the screen of the mobileterminal 100. The AR image includes a cleaner icon 920 and a travelingpath 930 and the cleaner icon 920 moves. As the cleaner icon 920 moves,the traveling path 930 of the cleaner is changed in real time.

FIG. 9D shows an example of displaying a cleaning region using an ARimage. The mobile terminal 100 may display the AR image of the cleaningregion. In this case, the AR image of the cleaning region may include acleaned region 940 and an uncleaned region 950. According to oneembodiment, the cleaned region 940 and the uncleaned region 950 may bedisplayed in the form of a figure having a predetermined area.

The AR image of the cleaning region may be displayed to be superimposedon the real image.

Referring to FIG. 9D, the AR image of the cleaning region displayed as avirtual image is displayed on the real image displayed on the screen ofthe mobile terminal 100. In this case, the AR image includes a cleanedregion 940 and an uncleaned region 950.

According to the present embodiment, the user can easily check thecleaned region and the uncleaned region by projecting the real spaceusing the camera of the mobile terminal 100.

Conventionally, the image of the real space and the image of thecleaning map are simultaneously displayed such that the user checks thecleaned region. In this case, since the two images are displayed with noprocessing applied, the two images are mixed, thereby decreasingreadability. Therefore, the user cannot clearly recognize the travelingpath and the cleaning map of the robot cleaner 10 corresponding to thereal space.

According to the present embodiment, the mobile terminal 100 displaysthe cleaning map to be superimposed on the real image using the ARimage. Since the cleaning map is displayed using the AR image, the usercan distinguishably recognize the AR image of the cleaning map and theimage of the real space. In addition, the user may clearly recognize thetraveling path in the real space.

FIGS. 10A to 10D are diagrams illustrating an example in which a mobileterminal according to an embodiment of the present invention displays anuncleaned region of a robot cleaner using an AR image.

The mobile terminal 100 according to the embodiment of the presentinvention may display the uncleaned region of the robot cleaner 10 usingthe AR image. Specifically, the mobile terminal 100 may request theinformation on the uncleaned region from the robot cleaner 10, when apredetermined region of the real image is displayed on the screencaptured by the built-in camera 121 in a state of activating the ARmode. In this case, the mobile terminal 100 may receive information onthe uncleaned region from the robot cleaner, map the uncleaned region ofthe real space to the uncleaned region of the virtual space and displaythe uncleaned region using the AR image.

Here, the real space may include the uncleaned region.

FIG. 10A shows notification of an uncleaned region. The mobile terminal100 may provide notification of an uncleaned region. Specifically, themobile terminal 100 may request information on the uncleaned region fromthe robot cleaner 10, when a predetermined region of a real space isincluded in the screen captured by the built-in camera 121.

When the information on the uncleaned region is received, the mobileterminal 100 may provide notification of the uncleaned region.Notification of the uncleaned region may be output in various formatssuch as image, text, audio, etc.

Referring to FIG. 10A, the mobile terminal 100 projects a predeterminedregion of the real space on the screen thereof. In this case, anotification message 1010 of the uncleaned region is displayed on thescreen of the mobile terminal 100. Therefore, the user may easily checkthat there is an uncleaned region in the real space projected by themobile terminal 100.

FIG. 10B shows an example of displaying an uncleaned region using an ARimage. The mobile terminal 100 may display an AR image of an uncleanedregion on the screen projected by the camera. In this case, the AR imageof the uncleaned region may be displayed along with the AR image of thecleaned region.

According to one embodiment, the AR image of the uncleaned region may beindicated by a dotted line having a predetermined trajectory and the ARimage of the cleaned region may be displayed by a solid line having apredetermined trajectory.

The mobile terminal 100 may display the AR image of the uncleaned regionand the AR image of the cleaned region to be distinguished from a realimage. For example, the mobile terminal 100 may display the AR image inthe form of a stereoscopic image, a holographic image or a moving image.

In addition, the mobile terminal 100 may display information on the ARimage in text form, along with the AR images.

Referring to FIG. 10B, a virtual AR image 1020 is displayed on the realimage displayed on the screen of the mobile terminal 100. In this case,the AR image 1020 includes a cleaned region 1020 a denoted by a solidline and an uncleaned region 1020 b denoted by a dotted line. Therefore,the user may check the uncleaned region.

FIG. 10C is another example of displaying an uncleaned region using anAR image. The mobile terminal 100 may display an AR image of anuncleaned region to be superimposed on a real screen projected by thecamera. In this case, the AR image of the uncleaned region may bedisplayed along with the AR image of the cleaned region. According toone embodiment, the AR image of the cleaned region and the AR image ofthe uncleaned region may be displayed in the form of a figure having apredetermined area. In this case, the AR image of the uncleaned regionmay be displayed in a figure denoted by a dotted line and the AR of thecleaned region may be displayed in a figure denoted by a solid line.

FIG. 10D shows another example of displaying an uncleaned region usingan AR image. The mobile terminal 100 may display only an AR image 1050of an uncleaned region. In this case, the AR image 1050 of the uncleanedregion may be displayed to be superimposed on a real image. Referring toFIG. 10D, an AR image 1050 of an uncleaned region displayed as a virtualimage is displayed on a real image displayed on the screen of the mobileterminal 100.

In general, the user does not monitor cleaning operation performed bythe robot cleaner 10 in real time. Accordingly, if the robot cleaner 10finishes cleaning but there is an uncleaned region due to an obstacle,the user should directly check the uncleaned region with eyes based on acleaning state.

According to the present embodiment, when the user activates the AR modeof the mobile terminal 100 to project the cleaning region, the AR imageof the uncleaned region is displayed. Therefore, the user may easily andintuitively check the uncleaned region.

FIG. 11 is a diagram showing a process of operating a mobile terminalaccording to an embodiment of the present invention.

The mobile terminal 100 starts an AR mode (S1101).

According to one embodiment, when the robot cleaner 10 in a dockingstate is included in the screen captured by the built-in camera 121, themobile terminal 100 may start the AR mode. In this case, even when therobot cleaner 10 disappears on the screen of the mobile terminal 100after activating the AR mode, the AR mode may be continuouslymaintained.

The AR image may include information on an object present in the realworld. Here, the object may include all types of things, animals orplants, persons, etc. configuring the real world, such as electronicdevices, components or parts, things or articles, buildings, etc.

In this case, information on the object may be acquired using a positionbased method (GPS based method), a marker based method, etc.

Specifically, in the position based method, the mobile terminal 100transmits GPS information and geomagnetic information (e.g., direction,slope information, etc.) to a server 300 and acquires information on anobject monitored by the mobile terminal 100 (e.g., an object captured bythe camera of the mobile terminal 100) from the server 300. In thiscase, the mobile terminal 100 may generate the acquired information onthe object using an AR image and superimpose the AR image on a realimage, thereby providing the information on the object.

In the marker based method, the mobile terminal 100 may find a markerdisplayed as an object, recognize the size of the marker and a distancefrom the mobile terminal 100, and determine a three-dimensional positionand/or distance of the object as the marker. The mobile terminal 100 maydirectly acquire the information on the object from the marker or mayacquire the information on the object associated with the marker fromthe server 300. In this case, the mobile terminal 100 may generate anddisplay the acquired information on the object using an AR image on animage or at a marker position. For example, the marker may includeinformation on the object in the form of a two-dimensional code or mayinclude a variety of data such as characters, numbers, symbols, controlcodes, etc. In this case, the mobile terminal 100 may decode the imageof the marker, in which the information on the object is encoded, atwo-dimensional code, etc. and acquire the information on the object. Adetailed method of configuring a marker using a two-dimensional code maybe understood as being similar to the known two-dimensional code (e.g.,QR code, PDF417, DataMatrix, MaxiCode, etc.) and thus a detaileddescription thereof will be omitted.

Alternatively, the marker may include or provide information used toacquire or access the information on the object. For example, theinformation may be a unique number composed of different numbers orcharacters per information on the object, URL information capable ofaccessing the information on the object, identification information(serial number of the marker, etc.) for identifying different markers.In this case, the mobile terminal 100 may decode the image of themarker, in which the information is encoded, the two-dimensional code,etc. and acquire the information. The mobile terminal 100 may search theserver 300 for the information and acquire the information on theobject.

The mobile terminal 100 projects the real space using the built-incamera (S1102).

In this case, the mobile terminal 100 may display the real spaceprojected by the camera on the screen. Here, the real space may includethe cleaning region in which cleaning is performed by the robot cleaner10. Therefore, the cleaning region is included in the screen projectedby the camera and such a screen may become a real image or background,on which an AR image is superimposed.

The mobile terminal 100 receives information on the traveling path fromthe robot cleaner 10 (S1103).

Specifically, the mobile terminal 100 may request information on thetraveling path from the robot cleaner 10 and receive the information onthe traveling path from the robot cleaner 10 in response thereto. Tothis end, the mobile terminal 100 may perform communication with therobot cleaner 10.

The mobile terminal 100 maps the traveling path of the real space to thetraveling path of the virtual space (S1104).

To this end, the mobile terminal 100 may map the two-dimensionalcoordinates of the real space of the robot cleaner 10 to thethree-dimensional coordinates of the virtual space to map the real spacewhere the robot cleaner 10 is located to the virtual space displayed onthe screen of the mobile terminal 100.

The mobile terminal 100 generates the AR image of the traveling path ofthe virtual space (S1105).

According to one embodiment, the AR image of the traveling path of thevirtual space may display at least one of the cleaning map, component,operation, function and state of the robot cleaner along with thetraveling path of the robot cleaner 10.

Here, the traveling path of the robot cleaner 10 may be a trajectory ofthe robot cleaner 10 moved for cleaning. The component may include themain body 11, the suction device 12, the suction nozzle 13, the dustcollector 14, the driving wheels 15 a, the motors 15 b, and theauxiliary wheel 15 c, etc. The operation or function may includecleaning, traveling, movement, dust collection function, cleaning time,cleaning region, cleaning strength, etc. The state may include acleaning state, a cleaned state, etc.

Such an AR image may be displayed in various shapes or colors accordingto properties of objects or content to be displayed. For example, the ARimage of the traveling path may be displayed in correspondence with thetraveling path and may be denoted by an arrow or a solid line dependingon whether cleaning is finished.

The mobile terminal 100 displays the generated AR image to besuperimposed on the real image (S1106).

The mobile terminal 100 may display the AR image on the real imageprojected by the camera. For example, the mobile terminal 100 maydisplay the AR image of the traveling path of the robot cleaner 10 to besuperimposed on the image of the real space.

In this case, the mobile terminal 100 may display the AR image to bedistinguished from the real image. For example, the mobile terminal 100may display the AR image in the form of a stereoscopic image, aholographic image or a moving image.

In addition, the mobile terminal 100 may display the information on theAR image in text form along with the AR image.

FIG. 12 is a diagram showing a process of operating a mobile terminalaccording to another embodiment of the present invention.

The mobile terminal 100 starts an AR mode (S1201).

According to one embodiment, when the robot cleaner 10 in a dockingstate is included in the screen captured by the built-in camera 121, themobile terminal 100 may start the AR mode. In this case, even when therobot cleaner 10 disappears on the screen of the mobile terminal 100after activating the AR mode, the AR mode may be continuouslymaintained.

The mobile terminal 100 projects the real space using the built-incamera (S1202).

In this case, the mobile terminal 100 may display the real spaceprojected by the camera on the screen. Here, the real space may includethe cleaning region in which cleaning is performed by the robot cleaner10. Therefore, the cleaning region is included in the screen projectedby the camera and such a screen may become a real image or background,on which an AR image is superimposed.

The mobile terminal 100 receives information on the uncleaned regionfrom the robot cleaner 10 (S1203).

Specifically, the mobile terminal 100 may request information on theuncleaned region from the robot cleaner 10 and receive the informationon the uncleaned region from the robot cleaner 10 in response thereto.To this end, the mobile terminal 100 may perform communication with therobot cleaner 10.

The mobile terminal 100 maps the uncleaned region of the real space tothe uncleaned region of the virtual space (S1204).

To this end, the mobile terminal 100 may map the two-dimensionalcoordinates of the real space of the robot cleaner 10 to thethree-dimensional coordinates of the virtual space to map the real spacewhere the robot cleaner 10 is located to the virtual space displayed onthe screen of the mobile terminal 100.

The mobile terminal 100 generates the AR image of the uncleaned regionof the virtual space (S1205).

In some embodiments, the mobile terminal may display the AR image of thecleaned region along with the AR image of the uncleaned region or maydisplay only the AR image of the uncleaned region. Accordingly, if theAR image of the uncleaned region and the AR image of the cleaned regionare simultaneously displayed, the mobile terminal 100 may generate theAR image of the cleaned region.

The mobile terminal 100 displays the generated AR image to besuperimposed on the real image (S1206).

The mobile terminal 100 may display the AR image of the uncleaned regionto be superimposed on the image of the region projected by the camera.For example, the mobile terminal 100 may display the AR image of theuncleaned region of the robot cleaner 10 to be superimposed on the imageof the real space.

In this case, the mobile terminal 100 may display the AR image to bedistinguished from the real image. For example, the mobile terminal 100may display the AR image in the form of a stereoscopic image, aholographic image or a moving image.

In addition, the mobile terminal 100 may display the information on theAR image in text form along with the AR image.

According to the present invention, a user can intuitively and clearlyrecognize a real space corresponding to the traveling path of a robotcleaner.

According to the present invention, a user can intuitively and clearlyrecognize a real space corresponding to an uncleaned region.

The present invention mentioned in the foregoing description may beimplemented as code that can be written to a computer-readable recordingmedium and can thus be read by a computer system. Examples of possiblecomputer-readable mediums include HDD (Hard Disk Drive), SSD (SolidState Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, a magnetictape, a floppy disk, an optical data storage device, the other types ofstorage mediums presented herein, and combinations thereof. If desired,the computer-readable medium may be realized in the form of a carrierwave (for example, a transmission over the Internet). The computer mayinclude the controller 180 of the mobile terminal.

The above exemplary embodiments are to be construed in all aspects asillustrative and not restrictive. The scope of the invention should bedetermined by the appended claims and their legal equivalents, not bythe above description, and all changes coming within the meaning andequivalency range of the appended claims are intended to be embracedtherein.

What is claimed is:
 1. A mobile terminal comprising: a camera; a displayconfigured to display a real image projected by the camera; and acontroller configured to recognize the real image to request a travelingpath corresponding to the real image from a robot cleaner, to generatean augmented reality (AR) image of the traveling path based on thetraveling path received from the robot cleaner, and to display the ARimage to be superimposed on the real image.
 2. The mobile terminalaccording to claim 1, wherein the AR image of the traveling pathincludes at least one of a cleaned region and an uncleaned region. 3.The mobile terminal according to claim 2, wherein, if the cleaned regionand the uncleaned region are included in the AR image of the travelingpath, the controller distinguishably displays the cleaned region and theuncleaned region.
 4. The mobile terminal according to claim 2, whereinthe controller denotes the cleaned region by a solid line and denotesthe uncleaned region by a dotted line.
 5. The mobile terminal accordingto claim 2, wherein, if the uncleaned region is included in the AR imageof the traveling path, the controller controls the display to display anotification message of the uncleaned region.
 6. The mobile terminalaccording to claim 1, wherein the controller displays the AR image ofthe traveling path in the form of at least one of a stereoscopic image,a holographic image and a moving image.
 7. The mobile terminal accordingto claim 1, wherein the real image includes a region where cleaning isperformed by the robot cleaner.
 8. The mobile terminal according toclaim 1, wherein the controller generates and displays the AR image tobe superimposed on the real image, when the robot cleaner in a dockingstate is projected onto the real image.
 9. An augmented reality systemcomprising: a robot cleaner configured to perform cleaning whileautomatically traveling; and a mobile terminal including a displayconfigured to display a real image projected by a built-in camera and acontroller configured to recognize the real image to request a travelingpath corresponding to the real image from a robot cleaner, to generatean augmented reality (AR) image of the traveling path based on thetraveling path received from the robot cleaner, and to display the ARimage to be superimposed on the real image.
 10. The AR system accordingto claim 9, wherein the AR image of the traveling path includes at leastone of a cleaned region and an uncleaned region.
 11. The AR systemaccording to claim 10, wherein, if the cleaned region and the uncleanedregion are included in the AR image of the traveling path, thecontroller distinguishably displays the cleaned region and the uncleanedregion.
 12. The AR system according to claim 10, wherein the controllerdenotes the cleaned region by a solid line and denotes the uncleanedregion by a dotted line.
 13. The AR system according to claim 9, whereinthe controller displays the AR image of the traveling path in the formof at least one of a stereoscopic image, a holographic image and amoving image.
 14. The AR system according to claim 9, wherein thecontroller generates and displays the AR image to be superimposed on thereal image, when the robot cleaner in a docking state is projected onthe real image.
 15. A recording medium having recorded thereon acomputer-readable program for performing a method of operating a mobileterminal, the method comprising displaying a real image projected by abuilt-in camera; recognizing the real image to request a traveling pathcorresponding to the real image from a robot cleaner; generating anaugmented reality (AR) image of the traveling path based on thetraveling path received from the robot cleaner; and displaying the ARimage to be superimposed on the real image.
 16. The recording mediumaccording to claim 15, wherein the AR image of the traveling pathincludes at least one of a cleaned region and an uncleaned region. 17.The recording medium according to claim 16, wherein the displaying ofthe AR image to be superimposed on the real image includesdistinguishably displaying the cleaned region and the uncleaned regionif the cleaned region and the uncleaned region are included in the ARimage of the traveling path.
 18. The recording medium according to claim15, wherein the generating of the AR image includes: mapping thetraveling path received from the robot cleaner to a traveling path of avirtual space; and generating the AR image of the traveling path in themapped virtual space.
 19. The recording medium according to claim 15,wherein the real image includes a region where cleaning is performed bythe robot cleaner.
 20. The recording medium according to claim 15,wherein the method further includes generating and displaying the ARimage to be superimposed on the real image, when the robot cleaner in adocking state is projected onto the real image.